Adding S3 Support for Artifacts #34
Conversation
Codecov ReportAttention:
Additional details and impacted files@@ Coverage Diff @@
## main #34 +/- ##
==========================================
- Coverage 77.64% 74.71% -2.94%
==========================================
Files 9 9
Lines 331 348 +17
==========================================
+ Hits 257 260 +3
- Misses 74 88 +14 ☔ View full report in Codecov by Sentry. |
There was a problem hiding this comment.
Thank you for your contribution!
Can you please make a couple of changes:
- break down logartifact into two underlying functions, one for local disk and one for S3 - and then call them respectively from logartifact based on `!startswith(artifact_uri, "s3://")
- instead of hardcoding /tmp, use
mktemp
There was a problem hiding this comment.
Thanks for your contributions, @usmcamp0811.
I want to know if having both AWS and AWSS3 is completely necessary. I mean, does not the AWS package do the same as the other one?
Please don't forget to add/edit tests to this change.
|
@pebeto all the docs I looked at for AWS said to use both the packages, but I can try to leave one off. @deyandyankov yea I can make those changes here soon. I'm pretty busy right now, catching up from the holidays but should be able to get to it this weekend 🤞 |
|
You can find Mocking.jl useful to write tests for this specific implementation. |
I have enhanced the
logartifactfunction to accommodate MLFlow instances configured with S3 storage for artifact management. This enhancement includes an added verification step to determine if the specified URI corresponds to an S3 bucket. In cases where the URI is not an S3 bucket, the function retains its original behavior of saving artifacts to the disk. Conversely, when an S3 bucket is identified, the function leverages AWS and Minio services for artifact storage in the bucket. This implementation supports both AWS and Minio, although its compatibility with AWS S3 buckets has not been verified due to the lack of an AWS S3 bucket for testing. However, extensive testing has been conducted with an MLFlow instance utilizing Minio S3 buckets, and the functionality aligns with expected outcomes. It's important to note that when using Minio or similar S3 services, theMLFLOW_S3_ENDPOINT_URLenvironment variable must be set to ensure proper operation.